Before releasing any set of computer code, software manufacturers test their code rigorously. Ideally, every set of code should be 100% tested prior to release so that once released, the computer code will not experience any errors or “bugs”. From a practical standpoint, however, this is extremely difficult if not impossible. To do so would require more resources (both in terms of human resources and actual costs) than would be practicable to expend. As a result, most manufacturers settle for a compromise. So long as the testing of a set of code has reached a certain threshold (e.g. 85% of the code has been executed and tested), the code is deemed ready for release.
To determine whether a set of code has been sufficiently tested, manufacturers generally run tests on the code, and generate test coverage data based upon the tests. Basically, the coverage data specifies which lines of source code were executed during the tests, and what percentage of the lines of source code were actually executed and tested. Based upon the coverage data, the manufacturer determines whether the set of code has been sufficiently tested.
There are several well-known methodologies for generating test coverage data. The current methodologies have some common characteristics that make them undesirable and sometimes impossible to use in certain implementations. One of these characteristics is the amount of storage that is consumed in generating the coverage data. As an example, it is not uncommon, especially for complicated sets of computer code, for the coverage data to consume several megabytes of storage. This is not a problem if the testing and the data generation are carried out on a large-scale system in which gigabytes of storage are available. However, for small-scale systems (e.g. personal digital assistants (PDA's), cellular phones, etc.) in which storage is significantly limited, this large storage consumption is a very significant problem. If the small-scale system does not have enough storage to accommodate both the computer code and the coverage data, then it effectively means that the computer code cannot be tested, and coverage data cannot be generated on that system. This poses a serious testing problem.
Another characteristic of current coverage data generation methodologies is that they significantly slow down the execution of the tests and the computer code. In some instances, generation of coverage data can cause execution of the tests and computer code to take several times longer than if the tests and computer code were executed without generating coverage data. The testing of computer code already is a time consuming process. To increase that time several fold would be an unacceptable result in many implementations.
As the above discussion illustrates, the current methodologies for generating test coverage data have significant drawbacks. As a result, an improved methodology is needed.